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Thirty-nine  researchers  attended  the  Workshop  on  Optical  Neural  Networks  held  in  Jack- 
son,  Wyoming  this  pcist  February  7-10,  1990.  The  Workshop,  sponsored  by  the  Air  Force 
Office  of  Scientific  Research  and  the  Office  of  Naval  Research,  was  organized  by  Lee  Giles 
(NEC  Research  Institute),  Demetri  Psaltis  (California  Institute  of  Technology),  and  Kelvin 
Wagner  (Optoelectronic  Computing  Systems  Center,  University  of  Colorado  at  Boulder).  Its 
purpose  was  to  critically  examine  the  status  of  optical  neural  network  research  and  evaluate 
its  present  and  future  role,  particularly  as  an  implementation  technology  for  neural  network 
models  of  computation. 

The  workshop  participants  included  researchers  in  optical  neural  networks  as  well  ais  ex¬ 
perts  in  related  fields  such  as  active  optical  devices,  VLSI  implementation  of  neural  networks, 
and  neural  network  architectures,  algorithms,  and  theory.  The  format  of  the  workshop  in¬ 
cluded  both  presentation  and  discussion  sessions.  Each  participant  had  prepared  a  15  minute 
presentation.  These  presentations  served  as  the  context  for  the  discussion  sessions.  A  brief 
summary  of  some  of  these  presentations  and  discussions  is  presented  below. 


1  Introduction 

Neural  networks  typically  consist  of  weighted  global  interconnections  between  arrays  of  sim¬ 
ple  nonlinear  units.  Their  output  is  usually  a  soft  threshold  version  of  the  weighted  and 
summed  inputs  from  other  neurons.  Learning  dynamics  are  used  to  evolve  the  interconnec¬ 
tion  weight  matrices  eis  a  succession  of  small  perturbations,  usually  implemented  as  sums  of 
outer  products.  These  are  the  essential  features  that  must  be  incorporated  into  any  hardware 
implementation  of  a  neural  network. 

Optical  techniques  are  being  considered  for  the  implementation  of  neural  network  mod¬ 
els  of  computation  because  of  several  unique  properties  of  optical  systems.  These  include 
the  three-dimensional  topology  of  optical  systems,  and  the  ability  of  optical  beams  to  cross 
through  one  another  in  free  space,  allowing  the  compact  implementation  of  global  intercon¬ 
nect  networks.  In  addition,  the  continuous  analog  nature  of  optical  systems  can  be  combined 
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Figure  1:  a:  The  spatial  light  modulator  based  approach  to  optical  neural  networks. 
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with  nonlinear  optical  devices  to  implement  nonlinear  dynamical  systems  which  are  a  good 
match  to  neurodynamical  models  of  computation  and  learning. 

Optical  approaches  to  the  implementation  of  neural  networks  are  usually  based  on  one 
of  two  distinct  techniques  for  implementing  the  weighted  interconnections  required  by  the 
neural  models.  The  first  approach,  shown  in  Figure  1,  uses  the  variable  transmittance  of 
a  pixel  in  a  two-  dimensional  spatial  light  modulator  (SLM)  to  represent  the  weight  of  an 
interconnection.  The  second  approach,  shown  in  Figure  2  uses  the  programmable  diffraction 
efficiency  of  a  holographic  grating  to  represent  the  weight  of  an  interconnection.  Both  tech¬ 
niques  rely  on  spatial  broadcasting  and  spatial  collection  of  the  weighted  outputs  to  complete 
the  required  matrix  vector  multiplication.  A  potential  advantage  of  the  holographic  tech¬ 
nique  that  was  emphasized  by  several  conference  participants  is  the  ability  to  utilize  volume 
holograms  to  store  the  interconnection  gratings  in  three  dimensions,  as  shown  in  Figure  2. 
This  allows  a  tremendous  density  of  weighted  interconnections  to  be  realized,  and  the  use  of 
dynamic  materials  (such  as  photorefractive  crystals)  allows  the  implementation  of  real-time 
learning  in  the  optical  domain.  This  is  based  on  an  extension  of  the  holographic  metaphor  for 
associative  memory  proposed  by  Van  Heerden  and  Gabor  more  than  20  years  ago.  Three  key 
developments  that  distinguish  modern  optical  neural  network  research  from  earlier  pioneer¬ 
ing  work  on  holographic  association  are  the  incorporation  of  dynamic  learning  algorithms, 
the  central  role  played  by  the  neural  nonlinearity,  and  the  utilization  of  fractal  topologies  in 
order  to  fully  realize  the  global  interconnection  capabilities  of  volume  holograms. 


2  Learning  Systems 

The  most  distinctive  feature  of  neural  network  models  of  computation  is  the  ability  to  learn 
from  experience.  This  is  accomplished  by  adaptively  modifying  the  strength  of  the  inter¬ 
connections  between  the  neurons.  In  optical  systems,  these  weights  are  usually  represented 
as  the  diffraction  efficiency  of  holographic  gratings  or  as  the  transmittance  of  pixels  in  a 
spatial  light  modulator.  Neural  learning  algorithms  give  rules  for  the  adaptive  modifica¬ 
tion  of  these  interconnections  or  weights,  which  are  almost  always  based  on  iterative  outer 
product  perturbations  of  the  weight  matrix.  This  can  be  mapped  into  optics  as  either  asso¬ 
ciative  holographic  recording  or  as  the  product  of  crossed  one-dimensional  light  modulators 
addressing  a  two-dimensional  optically  addressed  spatial  light  modulators. 

2.1  Photorefractive  perceptrons 

Several  successful  optical  learning  demonstrations  using  photorefractive  crystals  were  pre¬ 
sented  at  this  Workshop.  David  Brady  (University  of  Illinois,  Urbana-Champagne)  discussed 
the  limitations  of  controlling  dynamic  volume  holograms,  since  the  internal  degrees  of 
freedom  must  be  accessed  through  the  faces,  which  permit  the  addressing  of  only  inde¬ 
pendent  control  parameters.  In  neural  learning  applications,  outer  products  are  formed  with 
holographic  interference  between  patterns  sampled  on  appropriate  fractal  grids  that  modify 
all  internal  degrees  of  freedom,  but  not  independently.  A  sequence  of  exposures  must 
be  used  to  write  interconnection  matrices  into  a  volume  hologram,  and  exposure  scheduling 
needs  to  be  employed  to  maximize  the  diffraction  efficiency,  which  is  decreased  due  to  the 
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Figure  3:  Schematic  illustration  of  volume  hologram  association,  and  a  simple  learning 
system.  Input  output  pairs  on  the  fractal  grids  and  the  experimental  learning  curve.  Figure 
credit:  David  Brady,  Photorefractive  volume  holography  in  artificial  neural  networks,  PhD 
Thesis,  Caltech  1990. 
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Figure  4:  Schematic  diagram  of  the  perceptron  vyptical  pattern  classifier  using  a  photorefrac- 
tive  crystal.  Figure  Credit:  John  Hong,  Scott  Campbell,  and  Pochi  Yeh,  Optical  pattern 
classifier  with  perceptron  learning,  Applied  Optics,  vol  29(20),  p.  3019  (1990). 
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Experimental  learning  curves  for  five  training  pat¬ 
terns  (A,  B,  C,  D,  and  E). 


Figure  5:  Schematic  of  the  holographic  learning  machine  for  multicategory  classification. 
Figure  Credit:  Eung  Gi  Pack,  et  al.,  Optics  Letters,  Vol.  14  (23),  p.  1304,  1989. 
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effect  that  subsequent  exposures  partially  erase  earlier  ones.  He  also  showed  the  results  of 
a  photorefractive  perceptron  which  associated  500  input  random  patterns  with  500  pixel 
names,  shown  in  Figure  3,  where  the  weight  decrements  required  by  the  learning  algorithm 
were  implemented  using  incoherent  erasure. 

John  Hong  (Rockwell  Science  Center,  Thousand  Oaks,  CA)  demonstrated  the  successful 
operation  of  a  photoreceptive  perceptron.  His  system,  shown  in  Figure  4,  uses  coherent 
ercisure  to  decrease  the  adaptive  weights,  where  a  z  phase  shifted  grating  is  written  on  top  of 
an  existing  grating  using  a  double  mach-zehnder  interferometer,  thereby  partially  canceling 
the  initial  grating. 

Eung  Gi  Paek  (Bellcore,  New  Jersey)  also  presented  a  perceptron  learning  system  that 
uses  a  photorefractive  crystal  as  the  adaptive  weights.  The  system,  shown  schematically  in 
Figure  5,  had  250,000  input  neurons  and  10  bipolar  outputs.  His  research  demonstrated  that 
a  multiple  output  perceptron  could  be  implemented  by  multiplexing  gratings  in  a  volume 
medium  without  unwanted  crosstalk.  Both  Paek’s  and  Hong’s  systems  exhibited  anomalous 
unlearning  due  to  incoherent  erasure  in  the  crystals  after  the  desired  pattern  associations  had 
been  learned  and  were  simply  being  read  out.  This  may  be  a  severe  problem  for  the  optical 
implementation  of  adaptive  systems  using  photorefractive  holograms  unless  a  compensating 
learning  algorithm,  or  nondestructive  readout  technique  is  utilized. 

2.2  SLM  based  learning  systems 

Successful  learning  was  also  obtained  in  systems  employing  SLMs  as  the  adaptive  intercon¬ 
nections.  Kristina  Johnson  (OCS  Center,  CU-Boulder)  presented  results  of  single-layer  and 
multi-layer  learning  experiments  in  a  polarization  based  liquid  crystal  optical  connectionist 
machine.  (See  Figure  6.)  This  optical  connectionist  machine  has  performed  back  propaga¬ 
tion  learning  that  successfully  predict  solar  flare  activity  given  32  coded  input  and  3  labeled 
output  neurons  when  trained  on  200  patterns  of  sunspot  data.  The  learning  dynamics,  which 
were  controlled  by  the  feedback  computer,  successfully  compensated  for  several  varieties  of 
noise  due  to  imperfections  in  the  optics. 

Nabil  Farhat  (University  of  Pennsylvania)  presented  work  on  Boltzmann  machine  learn¬ 
ing  using  binary  spatial  light  modulators.  He  showed  how  a  multilayered  network  can  be 
implemented  in  a  single  layer  of  hardware  by  partitioning  the  weight  matrix  into  a  number  of 
blocks  representing  the  interconnections  between  different  layers  cis  shown  in  Figure  7.  Dr. 
Farhat  and  Dr.  Anderson  also  discussed  pheise  space  engineering  techniques  which  describes 
their  approach  to  designing  an  optical  neural  network.  This  entails  the  design  of  the  path  of 
a  complex  system  through  its  state  space  and  represents  a  computation  as  the  state  space 
evolution  of  the  system. 

3  Optical  Synapse  Technology 

The  successful  realization  of  optical  neural  networks  is  dependent  upon  the  availability  of 
components  that  can  act  as  neurons  and  synapses.  The  optical  synapses  must  weight  the 
interconnections  between  the  neurons.  The  ability  to  be  dynamically  modify  these  synapses 
is  required  in  order  to  implement  adaptive  learning  algorithms. 
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Figure  6;  The  polarization-baised  optical  connectioni.st  niachineal  Lite  Univorsily  of  Colorado 
is  pictured  here.  Figure  credit:  Kristina  Johnson 
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Figure  7:  Optoelectronic  analog  of  self-organizing  neural  net  partitioned  into  three  layers 
capable  of  stochastic  self-programming  and  learning.  Figure  credit:  Nabil  Farhat,  Applied 
Optics,  Vol.26  23),  p.  5097,  1987. 
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3.1  Dynamic  volume  holograms 

Holographic  interconnections  using  dynamic  volume  holograms  were  discussed  by  several 
participants.  A  major  difficulty  with  these  materials  is  the  incoherent  erasure  that  occurs 
while  the  holograms  are  being  read  out,  which  results  in  the  unlearning  phenomena  observed 
by  Paek  and  Hong.  This  incoherent  erasure  problem  in  photorefractives  can  be  alleviated 
by  inducing  a  read-write  asymmetry. 

Henri  Rajenbach  (Thomson-CSF,  France)  presented  preliminary  results  of  hole  fixing  in 
cooled  BSO  that  resulted  in  the  ability  to  continuously  read  out  images  for  several  hours 
without  erasure.  This  is  possible  because  a  photogenerated  hologram  written  by  the  electrons 
is  compensated  by  the  holes,  and  at  low  temperatures  the  hole  mobility  is  much  lower  than 
the  electron  mobility,  so  that  a  hologram  written  as  spatial  modulations  of  the  hole  density 
remains  frozen  into  the  crystal. 

Fai  Mok  (Northrup  Corp.  Research  and  Technology  Center,  Palos  Verdes,  CA)  pre¬ 
sented  results  of  multiple  image  storage  in  LiNb03:Fe  that  used  thermal  fixing  and  exposure 
scheduling  to  compensate  for  erasure  during  writing,  and  also  eliminates  erasure  during  read¬ 
out.  He  has  successfully  stored  and  retrieved  over  500  high  quality  images  of  over  60,000 
pixels  each,  with  greater  than  .01%  diffraction  efficiency  each.  The  holograms  were  recorded 
with  a  300:1  beam  ratio  at  an  angular  separation  of  .02  degrees,  and  exposure  scheduling 
was  used  in  order  to  obtain  a  diffraction  efficiency  uniformity  of  about  25%. 

Kelvin  Wagner  (University  of  Colorado,  Boulder  CO)  showed  samples  of  organic  volume 
holograms  based  on  photochemical  dyes  suspended  in  a  polymer  matrix  that  may  find  ap¬ 
plications  in  optical  neural  networks.  These  materials  exhibit  high  diffraction  efficiency  and 
show  both  permanent  and  dynamic  holographic  recording  capabilities  that  may  be  useful  for 
avoiding  unlearning  associated  with  erasure.  The  very  low  cost  and  large  size  may  give  them 
a  competitive  economic  advantage  over  the  more  costly  and  fragile  photorefractive  crystals 
commonly  employed. 

Dana  Anderson  (OCS  Center,  CU-Boulder)  showed  winner-take-all  behavior  in  sequential 
recall  dynamics  in  a  multi-crystal  photorefractive  resonator,  shown  in  Figure  8.  These  type 
of  winner-take-all  dynamics  give  the  optical  system  an  important  decision  making  capability 
that  forms  the  basis  of  a  wide  variety  of  unsupervised  learning  algorithms.  Dr.  Anderson 
believes  that  the  computation  dynamics  inherent  to  the  photorefractive  circuits  may  be 
applied  as  a  very  general  technique  for  obtaining  almost  any  dynamics  provided  one  has  a 
sufficient  number  of  modes  and  sufficient  control  over  the  mode  interactions.  Furthermore, 
this  class  of  optical  system  is  one  of  the  few  physical  systems  that  can  embed  continuous 
distributions  of  neural  activity  processing  in  continuous  time. 

3.2  Fixed  planar  holograms 

Art  Gmitro  (University  of  Arizona)  and  Henry  J.  White  (British  Aerospace)  both  presented 
analysis  of  the  limitations  of  space  variant  weighted  interconnections  using  E-Beam  written 
computer-generated  holograms  for  interconnecting  two-dimensional  neuron  arrays  displayed 
on  liquid  crystal  light  valves.  They  both  concluded  that  up  to  about  64x64  neuron  arrays 
could  be  globally  interconnected  in  order  to  implement  a  nonadaptive  optical  neural  network 
using  this  technique.  However,  larger  networks  would  be  beyond  the  technological  capabil- 
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Figure  8:  Optical  resonator  implementing  a  winner-take-all  dynamics.  Crystal  1  is  in  the 
Fourier  plane  of  the  vertically  multiplexed  fiber  array  while  crystal  2  is  in  the  image  plane, 
crystal  3  just  provides  gain.  Figure  credit:  Dana  Anderson 
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ities  of  this  technology.  This  is  because  the  size  and  resolution  requirements  of  a  space 
variant  interconnection  hologram  for  an  NxN  neuron  array  grow  as  K N^xK ,  where  K  is 
an  oversampling  factor,  typically  greater  than  10,  and  the  overall  space-bandwidth  product 
is  constrained  to  be  less  than  10^x10®  by  the  e-beam  machines.  In  addition,  due  to  various 
crosstalk  and  sidelobe  terms,  the  accuracy  of  these  weighted  interconnects  might  be  too 
low  for  some  applications  (such  as  neural  optimizers)  but  it  may  be  sufficient  for  optical 
cLSSociative  memory. 

Alan  Yamamura  (California  Institute  of  Technology)  presented  new  results  of  writing 
computer  generated  holograms  on  Sony’s  sampled-format  optical  disks.  Holographically 
reconstructed  weight  matrices  were  written  onto  optically  programmed  VLSI  neural  chips 
to  control  the  electronic  interconnect  topology  from  the  optical  domain.  This  allows  rapid 
reprogrammability  of  the  interconnections  in  order  to  implement  multilayer  networks. 


4  Optical  Neuron  Technology 

Simple  optical  neurons  need  to  sum  a  huge  number  of  weighted  inputs  and  produce  a  thresh¬ 
old  output.  The  goals,  stated  in  an  introduction  by  Demetri  Psaltis  (California  Institute 
of  Technology,  Pasadena),  are  to  produce  devices  with  mote  than  10,000  neurons  with  a 
response  time  in  the  microsecond  range,  with  gain  much  greater  than  10,  and  power  dissi¬ 
pation  well  below  a  milliwatt.  Although  such  a  device  capability  is  not  yet  available,  several 
technologies  appear  to  be  approaching  this  goal. 

4.1  FLC  modulators 

Ferroelectric  liquid  crystals  are  a  promising  candidate  for  implementing  large  arrays  of  low 
power  neurons.  Garret  Moddel  (OCS  Center,  CU-Boulder)  illustrated  the  capabilities  of  an 
amorphous  silicon  ferroelectric  liquid  crystal  (FLC)  optically-addressed  spatial  light  modu¬ 
lator  (OASLM)  using  smectic  C*  and  smectic  A*  materials.  The  smectic  A*  device  achieves 
CIS  low  ais  a  4  microsecond  response  time  for  a  high  resolution  SLM  (^701p/mm)  with  signifi¬ 
cantly  more  than  1000x1000  pixels,  requiring  significantly  below  .IpJ/pixel  switching  energy. 
These  capabilities  make  this  an  almost  ideal  device  for  implementing  optical  neurons.  Even 
further  improvements  can  be  made  by  including  a  third  terminal,  which  his  group  recently 
demonstrated,  allowing  them  to  implement  a  variable  threshold  device. 

Tim  Drabik  (Georgia  Institute  of  Technology,  Atlanta)  presented  recent  results  on  a 
silicon  VLSI  chip  coated  with  FLC  that  incorporated  a  16x8  array  of  photodetectors  where 
each  detector  drove  a  modulating  pad  containing  the  FLC  and  acted  as  an  optically-addressed 
SLM.  This  device  may  have  application  as  an  optical  neuron  array  or  as  an  early  vision  optical 
preprocessor.  He  also  showed  results  of  a  high  frame  rate  electrically  addressed  64x64  SLM 
that  might  be  used  for  entering  data  into  an  optical  neural  network. 

4.2  GaAs  modulators,  LEDS  and  laser  diodes 

Another  promising  approach  is  to  use  optoelectronic  GaAs  modulators  and  sources  such  as 
LEDs  and  laser  diodes  driven  by  electronic  photo-detectors.  Several  variants  were  proposed 
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for  optical  neurons,  and  their  properties  were  compared  and  contrasted. 

Tony  Lentine  (AT&T  Bell  Labs,  Naperville,  IL)  showed  results  of  a  1 .5pJ  switching  energy 
symmetric  self-  electro-optic-effect  device  (S-SEED).  Arrays  of  those  devices  can  implement 
loser- take-all  dynamics  by  electrically  interconnecting  them  in  series.  Although  this  is  the 
opposite  of  the  winner-take-all  dynamics  required  in  many  neural  models,  it  can  be  modified 
by  following  with  a  separate  optical  inversion  stage. 

Jcisprit  Singh  (university  of  Michigan,  Ann  Arbor)  suggested  that  SEEDs  were  not  suited 
to  neural  networks,  but  could  be  modified  by  placing  the  seed  inside  of  a  heterojunction 
bipolar  transistor,  rather  than  inside  a  PIN  diode  as  conventionally  done.  Not  only  could 
this  allow  the  utilization  of  the  HBT-SEED  as  a  neuron,  but  it  might  also  be  used  as  an 
adaptive  synapse  mask. 

J.  Cheng  (University  of  New  Mexico,  Albuquerque)  presented  arguments  in  favor  of  the 
use  of  micropower  laser  diodes  and  vertical  cavity  surface  emitting  lasers  as  the  sources  for 
optical  neurons.  This  is  motivated  by  the  excellent  optical  properties  and  diffraction  limited 
performance  of  these  sources,  as  well  as  by  the  high  differential  quantum  efficiency  that  can 
be  achieved.  He  suggested  that  a  photodarlington  ving  each  microlaser  could  provide  the 
necessary  optical  neuron  functions. 

Demetri  Psaltis  showed  results  of  a  10x10  array  of  phototransistor  darlington  amplifier 
LED  neuron  array  fabricated  in  GaAs.  LEDs  were  chosen  instead  of  more  efficient  diodes 
because  they  have  no  current  threshold  and  the  power  dissipation  of  a  very  large  array  of 
optical  neurons  can  be  minimized.  His  arguments  go  as  follows,  the  total  output  power  of 
an  array  of  N  laser  diodes  is  given  by 

Po%al  =  Nrj{i-  ith)hulq 

where  77  .3  —  .7  is  the  external  differential  quantum  efficiency  and  includes  the  resonator 

and  loss  effects,  i  is  the  input  current,  while  ith  ss  1mA  is  the  threshold  current  for  a  low 
threshold  laser.  Similarly  the  total  output  power  of  an  array  of  N'  LEDs  is  given  by 

Po‘'Xl  =  NW.'hr/q 

where  77'  %  .01  is  the  efficiency  for  an  LED,  and  i'  is  the  drive  current.  For  both  the  laser 
and  LED  the  dissipation  is  given  by  Pdi,  =  iv  and  the  drive  voltage  for  both  must  be  above 
the  bandgap.  For  a  fixed  power  requirement  as  the  number  of  neurons  is  increased,  the  total 
efficiency  of  the  lasers  drop  since  they  are  operated  closer  to  threshold,  while  for  the  LEDs 
the  efficiency  remains  constant.  Therefore,  for  more  than  about  1000  neurons  emitting  a 
few  mW,  the  LEDs  are  preferable.  However,  if  Watts  of  optical  power  is  required,  then  laser 
diodes  would  have  significantly  better  overall  efficiency. 

5  Implications  of  Neural  Network  Theory 

Neural  network  theory  has  several  implications  on  the  requirements  and  limitations  of  optical 
neural  networks.  These  issues  were  discussed  at  length  during  the  Workshop.  Two  specific 
consequences  are  mentioned  here. 
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5.1  dynamic  range  requirements  on  synapses 

Santosh  Venkatesh  (University  of  Pennsylvania)  presented  results  on  the  required  dynamic 
range  of  synapses  and  showed  that  with  binary  synapses  storage  capacity  is  only  reduced 
by  a  factor  of  two  from  real  synapses.  He  presented  a  simple  directed  drift  learning  al¬ 
gorithm  for  these  binary  synapses  and  concluded  that  low  dynamic  range  optical  synapses 
may  be  adequate  for  many  applications.  This,  however,  did  not  alleviate  the  dynamic  range 
requirements  of  the  neurons,  nor  did  it  address  the  issue  of  contrast  ratio  or  noise. 

5.2  Issues  in  training  and  generalization 

Eric  Baum  (NEC  Research  Institute,  Princeton,  NJ)  discussed  the  limitations  inherent  to 
learning  from  examples  and  concluded  that  the  huge  number  of  interconnects  available  in  op¬ 
tical  systems  may  require  a  huge  number  of  training  exemplars  in  order  to  achieve  valid  gen¬ 
eralization.  One  suggestion  to  alleviate  the  unfavorable  scaling  properties  of  huge  networks 
was  to  grow  or  prune  the  networks  to  fit  the  task.  He  suggested  that  the  superabundance  of 
resources  in  optics  may  allow  fast  learning,  followed  by  pruning,  to  achieve  a  minimal  optical 
network  quickly  and  obtain  good  generalization. 

John  Denker  (AT&T  Bell  Laboratories,  Holmdel  NJ)  discussed  the  difference  between 
capacity  and  generalization  in  a  neural  network,  and  illustrated  the  importance  of  massively 
exceeding  capacity  to  insure  valid  generalization.  Since  the  type  of  real  world  pattern  recog¬ 
nition  problems  likely  to  be  processed  on  an  optical  neural  network  are  somehow  structured 
in  a  very  high  dimensional  space  and  manifestly  not  random,  the  capacity  arguments  invoked 
for  randomly  selected  patterns  should  be  applied  cautiously. 

6  Some  Conclusions 

The  issues  brought  to  light  are  challenging  ones.  In  face  of  rapid  developments  in  electronic 
technologies,  are  optical  neural  networks  truly  viable?  The  discussions  were  lively,  and 
opinions  were  frankly  expressed  by  the  participants,  although  universal  agreement  Wcis  not 
achieved. 


6.1  Comparison  with  VLSI  neural  networks 

A  major  topic  of  discussion  was  the  viability  of  optical  neural  networks  in  light  of  the 
incredible  advances  foreseen  in  the  electronic  implementation  of  neural  networks.  Larry 
Jackel  (AT&T  Bell  Labs,  New  Jersey)  and  Josh  Allspector  (Bellcore,  New  Jersey)  illustrated 
the  great  potential  of  the  VLSI  approach  to  neural  networks  with  numerous  examples  of 
fabricated  chips.  However,  VLSI  neural  networks  are  limited  to  two  dimensions  while  optics 
can  make  use  of  the  three-dimensional  storage  capacity  of  volume  holograms  which  allows 
the  compact  realization  of  huge  networks  with  N310,000  neurons,  well  beyond  the  projected 
capabilities  of  VLSI.  The  trainability  of  such  networks  and  the  search  for  problems  that 
inherently  require  these  large  networks  are  still  open  questions. 
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6.2  Fault  tolerance 


Several  researchers  emphcisized  that  the  optical  implementations  of  adaptive  networks  are 
capable  of  learning  out  some  of  the  imperfections  inherent  to  optical  systems.  Because  of  this 
these  systems  may  stand  a  better  chance  for  success  than  other  optical  computing  paradigms 
since  they  are  inherently  more  fault  tolerant  and  robust. 

6.3  Biology  and  neural  network  theory 

The  importance  of  biological  principles  was  emphasized  by  several  researchers  since  the  brain 
provides  an  existence  proof  of  the  utility  of  the  neural  paradigm.  There  is,  however,  currently 
a  mismatch  between  theoretical  models  and  optical  implementation  because  many  of  the 
models  are  derived  directly  from  biological  principles  or  intended  for  VLSI  Implementation 
rather  than  optical,  and  a  closer  coupling  between  optics  and  neural  network  theory  was 
suggested. 

6.4  Applications  of  optical  neural  networks 

Many  discussions  centered  around  applications  appropriate  for  optical  neural  nets.  Image 
recognition,  an  inherently  two-dimensional  problem,  seem  to  be  a  natural  match  for  optics, 
especially  if  the  shift  invariant  properties  of  Fourier  optical  systems  can  be  combined  with 
some  recent  neural  models  that  require  correlation.  Problems  such  as  AI  data  base  searches 
may  also  be  able  to  take  advantage  of  the  huge  storage  capacity  and  dimensionality  of  optical 
systems.  Naturally  dynamical  problems  such  as  speech  and  radar  may  find  ideal  matches  in 
nonlinear  dynamical  optical  systems. 

A  final  conclusion  of  the  workshop' was  that  optical  neural  networks  need  to  be  applied 
to  a  large-scale  application  problem  well  beyond  the  capabilities  of  electronics  in  order  to 
demonstrate  the  viability  of  this  nonconventional  approach  to  computation. 

7  Future  Plans 

The  workshop  was  quite  successful  as  a  vehicle  leading  to  important  discussions,  critical  self 
evaluation,  and  directing  future  research  in  the  most  advantageous  and  necessary  directions. 
A  follow  on  workshop  may  be  held  in  early  1992. 
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WORKSHOP  ON  OPTICAL  NEURAL  NETWORKS 


February  7-10,  Jackson  WY 


39  attendees  from  the  US,  UK,  France,  and  West  Germany 
3  I  day  workshop  emphasizing  discussion 

Presentation  Sessions 

Complexity  and  Limitations  of  Neural  Networks 
Optical  Neural  Network  Architectures 
Optical  Learning 
Learning  Theory  , 

Optoelectronic  Neural  Networks 

Holographic  Optical  Neural  Networks 

Devices  for  Optical  Neurons 

Devices  for  Optical  Synapses 

VLSI  Networks  Compared  to  Optics 

Optical  Interconnects 

Applications  of  Optical  Neural  Networks 

Discussion  Sessions 

Accuracy,  Scalability,  and  Noise 
Off-line  vs  Real-time  learning 
Learning 

2D  vs  3D  implementations 

Required  Technological  Advances 

LEDs  vs  Lasers  vs  Modulators  as  Optical  Neurons 

VLSI  vs  Optical  vs  Hybrid  Implementations  of  Neural  Networks 

Applications  of  Optical  Neural  Networks 

Advantages  and  Limitations  of  Optical  Neural  Networks 


SCHEDULE  -  Jackson  workshop  on  Optical  Neural  Networks,  Feb  7-10 


Wednesday,  February  7  Evening  session 


6:00-7:00 

Dinner 

7:30-7:45 

Organization  and  Welcome  D.  Psaltis 

7:45-8:45 

Complexity  and  Limitations  of  NN  -  John  Denker 

7:45-8:05 

S.  Venkatesh 

8:05-8:25 

E.  Baum 

8:25-9:00 

Discussion  of  accuracy,  scalability,  and  noise 

9:00-9:40 

Optical  Neural  Network  Architectures  -  Dana  Anderson 

9:00-9:20 

N.  Farhat 

9:20-9:40 

K.  Wagner 

9:40-10:00 

Discussion  of  why  optics 

10:00  - 

Snacks 

Thursday,  February  8 

Morning  session 

7:00-8:00 

Buffet  breakfast 

8:00-9:00 

Optical  learning  -  Kelvin  Wagner 

8:00-8:20 

E.  Paek 

8:20-8:40 

K.  Johnson 

8:40-9:00 

D.  Brady 

9:00-9:20 

J.  Hong 

9:00-9:45 

Discussion  of  off-line  vs  real-time  vs  computerized  learning 

9:45-10:15 

Coffee  Break 

10:15-11:30 

Learning  Theory  -  Larry  Jackel 

10:15-10:35 

J.  Denker 

10:35-10:55 

L.  Giles 

10:55-11:30 

Discussion  of  learning 

11:30 

Lunch 

Evening  session 

6:00-7:00 

Dinner 

7:00-7:40 

Optoelectronic  neural  networks  -  Nabil  Farhat 

7:00-7:20 

G.  Haussler 

7:20-7:40 

A.  Yamamura 

7:40-9:15 

Holographic  Optical  neural  nets  -  Demetri  Psaltis 

7:40-8:00 

D.  Anderson 

8:00-8:15 

E.  Hartman 

8:15-8:30 

Break 

8:30-8:45 

H.J.  White 

8:45-9:00 

A.  Gmitro 

9:00-9:15 

H.  Rajenbach 

9:30-10:00 

Discussion  of  2D  vs  3D  implementations 

10:00- 

Snacks 
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Friday,  February  9 

Morning  session 

7:00-8:00 

Buffet  breakfast 

8:00-9:30 

Devices  for  optical  neurons  -  Kristina  Johnson 

8:00-8:15 

S.  Esener 

8:15-8:30 

D.  Psaltis 

8:30-8:45 

J.  Cheng 

8:45-9:00 

J.  Singh 

9:00-9:15 

T.  Lentine 

9:15-9:30 

T.  Drabik 

9:30-10:00 

Coffee  Break 

10:00-10:45 

Devices  for  adaptive  optical  synapses  -  Josh  Allspector 

10:00-10:15 

G.  Moddel 

10:15-10:30 

C.  Warde 

10:30-10:45 

F.  Mok 

10:45-11:30 

Discussion  of  required  technological  advances 

11:30 

Lunch 

Evening  session 

5:30-7:30 

Dinner  at  the  Cadillac  Grill 

7:30-8:30 

VLSI  neural  networks  compared  to  optics  -  Sing  Lee 

7:30-7:50 

L.  Jackel 

7:50-8:10 

J.  Allspector 

8:10-8:30 

K.  Kornfield 

8:30-9:30 

Optical  interconnects  -  Sadik  Esener 

8:30-8:50 

M.  Prise 

8:50-9:10 

D.  Chiarulli 

9:10-9:30 

S.  Levitan 

9:30-10:00 

Discussion  of  VLSI  vs  digital  vs  optics  vs  hybrid  implementations 

10:00- 

Snacks 

Saturday,  February  10 

Morning  session 

7:00-8:00 

Buffet  breakfast 

8:00-9:00 

Optical  interconnects  2  -  Bill  Miceli 

8:00-8:20 

S.  Lee 

8:20-8:40 

P.  Lalanne 

8:40-9:00 

C.  Slinger 

9:00-10:00 

Applications  of  ONN  -  Lee  Giles 

9:00-9:15 

W.  Miceli 

9:15-9:30 

J.  Yu 

9:30-9:45 

A.  Craig 

9:45-10:00 

Coffee  Break 

10:00-10:30 

Discussion  of  appropriate  algorithms  and  applications  for  ONN 

10:30-11:00 

Discussion  of  advantages  and  limitations  of  ONN 

11:00-11:30 

Summary  and  Conclusions 

11:30 

Lunch  and  Adjourn 
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Successful  Demonstrations  of  Optical  Learning 
E.  Paek,  Bellcore 

250,000  inputs,  10  outputs,  dual  rail  bipolar 

Perceptron  learning  in  photorefractive  using  electronic  feedback 

Problems  with  erasure  in  photorefractives 


K.  Johnson,  University  of  Colorado  OCS 

32  inputs,  32  outputs  -  single  layer  with  electronic  learning 
17  inputs,  10  hidden,  3  outputs  -  electronic  backprop 
Learning  weights  as  polarization  rotating  pixels  on  LC  SLM 
Learning  successfully  compensated  for  optical  imperfections 


Compact,  spatially  multiplexed  FLC  OCM 


J.  Hong,  Rockwell  Sciences 

Image  classification  using  a  photorefractive  perceptron 
Coherent  erasure  by  Stokes  theorem 
Photorefractive  erasure  leads  to  unlearning 

F.  Mok,  Northrup 

Multiple  image  storage  in  photorefractive  LiNbOs 
500  images  with  no  crosstalk 

D.  Brady,  University  of  Illinois,  Urbana 

Optical  learning  in  volume  holograms,  sampling  grids 
Scheduling  for  learning,  read-write  assymetry,  loading 
500  input,  500  output,  learned  20  associations 
Perceptron  learning  demonstrated 

INPUT 


Optical  Synapse  Technology 

Photorefractive  Crystals 

Read- Write  assymetry  necessary  for  learning 
Cr  doped  BaTiOa  -  difficult  to  erase  gratings 
Hole  fixing  in  cooled  BSO 

500  holograms  (320x240)  angularly  multiplexed  in  LiNbOa'.Fe 
Thermal  fixing,  exposure  scheduling,  25%  uniformity 
.02°  separation,  300:1  beam  ratio 

Slow  rotation  to  eliminate  fanning  noise  in  photorefractive  amplifiers 
Photorefractive  fiber  sensor  for  robot  arm 

Winner-take-all  and  sequential  dynamics  in  photorefractive  resonators 

Fractal  sampling  grids  allow  global  interconnections 
Loading  of  holographic  degrees  of  freedom  requires  scheduled  training 
only  have  2iV^  control  parameters 

Holographic  Associative  Memories 

Space  Variant  Interconnections  with  CGH  very  inefficient 
NxN  array  of  neurons  requires  SN^xSN^  e-beam  CGH 
low  precision  interconnection  strengths,  use  redundancy 
N=64,  N^=4096,  hologram  =(6.4cm)^  with  Ifim  pixels 

Optical  disks  to  store  synaptic  weights 

CGHs  can  be  written  on  Sony  sampled  format  optical  disk 
Holographic  reconstruction  writes  weights  onto  VLSI  neural  network 
Uses  optically  controlled  synapses  on  an  electrical  neural  net  chip 


Optical  Neuron  Technology 

Sources  (LEDs  or  LDs)  vs  Modulators 
Silicon  hybrid  or  GaAs  monolithic 

Goals 

>10^  neurons  per  chip 
response  time  1msec  — >  1/isec  ^ 
optical  gain  10  — ^  10® 

power  dissipation  <C  ImW/neuron 

Technologies 

Photodetectors  and  Amplifiers 

Photodarlingtons,  photodiode  FETs,  ... 

Smart  Spatial  Light  Modulators  on  Silicon  VLSI 
PLZT 

ferroelectric  liquid  crystal 
GaAs  Source  -  Mutually  incoherent 
LEDs 

Surface  emitting  vertical  cavity  Laser  Di'^des 
Surface  emitting  microresonator  laser  diodes  -  optically  pumped 
GaAs  Modulator  Arrays 

SEEDs,  S3mimetric-SEED  arrays,  and  Multi-Seeds 
Hetero junction  Bipolar  SEEDs 
Optically  Addressed  Spatial  Light  Modulators 

H:a-Si  photodiodes  -  ferroelectric  liquid  crystal  SLMs 
Photorefractive  2-beam  coupling 

Resonator  dynamics  for  winner-take-all 


Implications  of  Neural  Network  Theory 


Learning  from  examples 

Need  enough  examples  to  learn  a  functional  mapping 

^  64W  ,  64iV 

m  >  - log - 

€  € 

m=^  examples  required  for  confident  generalization 
W  =  ^  weights,  N=  7^  neurons,  1-|  correctly  classified 
Optics,  huge  number  of  interconnections, 

therefore  huge  number  of  examples  are  required 

How  much  dynamic  range  do  synapses  need? 

Cover  capacity  =  2N,  with  binary  weights  capacity  =  N 
Binary  synapse  learning  with  directed  drift  algorithm 
Neurons  require  accuracy  1/N 
Low  dynamic  range  synapses  may  be  good  enough 

Training  restricted  neural  networks 

Domain  of  learning  must  be  constrained  or  learning  is  intractable 
Curse  of  dimensionality 

Higher  order  neural  networks 

Can  be  used  for  learning  grammars  and  sequential  processing 
Scales  very  poorly  in  VLSI,  N  inputs  per  neuron,  not  1  summed  input 
Optics  can  do  quadratic  elegantly 

Scalability 

Need  to  grow  or  prune  networks  to  fit  the  task 
Optics  has  a  superabundance  of  resources 
may  allow  fast  learning  then  pruning 


Some  Conclusions 


Optical  vs  Electronic  neural  networks 

Incredible  advances  in  electronic  neural  nets  are  foreseen. 

Optics  is  shooting  at  a  moving  target. 

Main  advantage  of  optics  is  high  connectivity 
Abihty  to  build  huge  networks,  N>  10,000 
What  problems  inherently  require  such  huge  networks? 

Can  such  huge  networks  be  trained? 

Require  enormous  number  of  training  samples 

3D  interconnections  using  volume  holograms 

Unique  to  optics 
The  brain  is  3D, 

VLSI  is  limited  to  2D. 

Allows  compact  reahzations  of  huge  networks. 

Optical  implementation  of  adaptive  nets 

Capable  of  learning  out  some  imperfections 

Stand  a  better  chance  for  success  than  other  optical  computing  paradigms. 

Mismatch  between  theoretical  models  and  optics. 

Develop  new  algorithms  matched  to  optical  hardware. 

Importance  of  biological  principles. 

Existence  proof  of  the  utility  of  the  neural  paradigm 
Holographic  style  processing  has  no  direct  electronic  counterpart 

Applications  appropriate  for  optical  neural  networks. 

Image  Recognition,  inherently  2D  problem. 

Take  advantage  of  shift  invariance. 

Naturally  dynamical  problems  such  as  speech 

Nonlinear  dynamics  can  be  matched  to  optical  physical  dynamics. 
Huge  problems 

AI  data  base  search 

Optics  needs  to  solve  a  real  problem 

One  that  electronics  can  not  do,  in  order  to  demonstrate  its  viabiUty. 
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What  can  optics  do  for  neural  networks?  This  was  the  question 
addressed  by  39  researchers  who  attended  the  Workshop  on  Optical 
Neural  Networks  in  Jackson,  WY,  Feb.  7-10,  1990.  The  workshop, 
sponsored  by  the  US  Air  Force  Office  of  Scientific  Research  and  the 
Office  of  Naval  Research,  was  organized  by  Lee  Giles  (NEC  Research 
Institute,  Princeton,  NJ),  Demetri  Psaltis  (California  Institute  of  Tech¬ 
nology  [CalTech],  Pasadena,  CA),  and  Kelvin  Wagner  (Optoelectroruc 
Computing  Systems  [OCS]  Center,  University  of  Colorado  [CU]  at 
Boulder).  .  , 

The  purpose  of  the  workshop  was  to  critically  examine  the  status  of 
optical  neural-network  research  and  evaluate  its  present  and  future 
roles,  particularly  as  an  implementation  technology  for  neural- 
network  models  of  computation.  Participants  included  researchers  in 
optical  neural  networks  o  dl  as  experts  in  related  fields  such  as  . 
active  optical  devices,  VLSI  implementation  of  neural  networks,  and 
neural-ne‘..crk  architectures,  algorithms,  and  theory. 

Neural  networks  typically  consist  of  weighted  global  interconnec-j  > 
tions  between  arrays  of  simple  nonlinear  units.  Their  output  is  usually  a 
soft  threshoiu  version  o‘  the  weighted  and  summed  inputs  firom  other 
neurons.  Learning  dynamics  are  used  to  evolve  the  interconnection 
weight  matrices  as  a  succession  of  small  perturbations,  usually 
implemented  as  sums  of  outer  products.  -These  are  the  essential  features 
that  must  be  incorporated  into  any  hardware  implementation  of  a 
neural  network. 

Optical  techniques  are  being  considered  for  the  implementation  of 
neural-network  models  of  computation  because  of  several  unique 
properties  of  optical  systems.  These  include  the  three-dimensional  (3- 
D)  topology  of  optical  systems  and  the  ability  of  optical  beams  to  cross 
through  one  another  in  free  space,  allowing  the  compact  implementa¬ 
tion  of  global  interconnect  networks.  In  addition,  the  continuous  analog 
nature  of  optical  systems  can  be  combined  with  nonlinear  optical 
devices  to  implement  nonlinear  dynamical  systems,  which  are  a  good 
match  to  neurodynamical  models  of  computation  and  learning. 

Optical  approaches  to  the  implementation  of  neural  networks  are 
usually  based  on  one  of  two  distinct  techniques  for  implementing  the 
weighted  interconnections  required  by  the  neural  models.  The  first 
approach,  shown  in  Fig.  1,  uses  the  variable  transmittance  of  a  pixel  in  a 
two-dimensional  (2-D)  spatial  light  modulator  (SLM)  to  represent  the 
weight  of  an  interconnection.  The  second  approach  uses  the  program¬ 
mable  diffraction  efficiency  of  a  holographic  grating  to  represent  the  . 
weight  of  an  interconnection.  Both  techniques  rely  on  spatial  broadcast¬ 
ing  and  spatial  collection  of  the  weighted  outputs  to  complete  the 
required  matrix  vector  multiplication.  -  V 

A  potential  advantage  of  the  holographic  technique,  which  was  '  . 
emphasized  by  several  conference  participants,  is  the  ability  to  utilize', 
volume  holograms  to  store  the  interconnection  gratii  gs  in  three  v 
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dimensions,  as  shown  in  Fig.  2.  This  allows  a 
tremendous  density  of  weighted  interconnections  to 
be  realized,  and  the  use  of  dynamic  materials  (such 
as  photorefractive  crystals  or  organic  holograms) 
allows  the  implementation  of  real-time  learning  in 
the  optical  domain.  This  is  based  bn  an  extension  of 
the  holographic  metaphor  for  associative  memory 
proposed  by  Van  Heerden  and  Gabor  more  than  20 
years  ago.  Two  key  developments  that  distinguish 
modem  optical  neural-network  research  from  earlier 
pioneering  work  on  holographic  association  are  the 
incorporation  of  dynamic  learning  algorithms  and 
the  central  role  played  by  the  neural  nonlinearity. 

Learning  systems 

The  most  distinctive  feature  of  neural-network 
models  of  computation  is  the  ability  to  learn  from 
experience.  This  is  accomplished  by  adaptively 
modifying  the  strength  of  the  interconnections  be¬ 
tween  the  neurons.  In  optical  systems,  these 


FIGURE  1 .  Controllable  pixel  transmittance  allows  optical  light 
modulator  to  work  in  an  optical  neural  network. 


weights  are  usually  represented  as  the  diffraction 
efficiency  of  holographic  gratings  or  as  the  transmit¬ 
tance  of  pixels  in  a  SLM.  Neural  learning  algorithms 
give  rules  for  the  adaptive  modification  of  these 
interconnections  or  weights,  which  are  almost  al¬ 
ways  based  on  iterative  outer  product  perturbations 
of  the  weight  matrix.  This  can  be  mapped  into  optics 
as  either  associative  holographic  recording  or  as  the 
product  of  crossed  one-dimensional  light  modulators 
addressing  a  2-D  optically  addressed  SLM. 

Several  successful  optical  learning  demonstra¬ 
tions  using  photorefractive  crystals  were  presented  at 
this  workshop.  John  Hong  (Rockwell  Science  Cen¬ 
ter,  Thousand  Oaks,  CA)  demonstrated  the  success¬ 
ful  operation  of  a  photoreceptive  perceptron.  His 
system  uses  coherent  erasure  to  decrease  the  adap¬ 
tive  weights,  where  a  tt  phase-shifted  grating  is 
written  on  top  of  an  existing  grating,  thereby 
partially  canceling  the  grating.  Eung  Gi  Paek  (Bell¬ 
core,  Livingston,  NJ)  presented  a  perception  learning 


system  that  uses  a  photorefractive  crystal  to  provide 
the  adaptive  weighting.  The  system  had  250,000 
input  neurons  and  10  bipolar  outputs.  His  research 
demonstrated  that  a  multiple-output  perceptron 
could  be  implemented  by  multiplexing  gratings  in  a 
volume  medium  without  unwanted  crosstalk.  Both 
Paek's  and  Hong's  systems  exhibited  anomalous 
unlearning  due  to  incoherent  erasure  in  the  crystals 
after  the  desired  pattern  associations  had  been 
learned  and  were  simply  being  read  out. 

Successful  learning  was  also  obtained  in  systems 
employing  SLMs  as  the  adaptive  interconnections. 
Kristina  Johnson  (OCS  Center,  CU-Boulder)  pre¬ 
sented  results  of  single-layer  and  multilayer  learning 
experiments  ih  a  polarization-based  liquid-crystal 
optical  connectionist  machine  (see  Fig.  3.).  "The 
optical  cormectionist  machine  has  performed  associ¬ 
ations  between  32  input  and  output  neurons  trained 


FIGURE  2.  Holographic  approach  to  optical  neural  networks 
takes  advantage  of  volume  Iwlograms.  '< 


on  200  patterns  of  sunspot  data  to  predict  solar  flare 
activity.  The  learning  dynamics  successfully  compen¬ 
sated  for  several  varieties  of  noise  due  to  imperfec¬ 
tions  in  the  optics. 

Nabil  Farhat  (University  of  Pennsylvania,  Philadel¬ 
phia,  PA)  presented  work  on  Boltzmarm  machine 
learning  using  binary  SLMs.  He  showed  how  a 
multilayered  network  can  be  implemer}ted  in  a  single 
layer  of  hardware  by  partitioning  the  weight  matrix 
into  a  number  of  blocks  representing  the  interconnec¬ 
tions  between  different  layers.  He  also  discussed 
phase-space  engineering,  which  describes  his  ap¬ 
proach  to  designing  an  optical  neural  network.  This 
entails  the  design  of  the  path  of  a  complex  system 
through  its  state  space  and  represents  a  computation 
as  the  state  space  evolution  of  the  system. 

Optical-synapse  technology 

The  successful  realization  of  optical  neural  net- 
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works  is  dependent  upon  the  availability  of  compo¬ 
nents  that  can  act  as  neurons  and  synapses.  The 
optical  synapses  must  weight  the  interconnections 
between  the  neurons.  The  ability  to  dynamically 
modify  these  synapses  is  required  to  implement 
adaptive-learning  dgorithms. 

Holographic  interconnections  using  dynamic  vol- 
'  ume  holograms  were  discussed  by  several  partici¬ 
pants.  A  major  difficulty  with  these  materials  is  the 
incoherent  erasure  that  occurs  while  the  holograms 
are  being  read  out,  which  results  in  the  unlearning 
phenomena  observed  by  Paek  and  Hong.  This 
incoherent  erasure  problem  in  photorefractives  can 
be  alleviated  by  inducing  a  read-write  asymmetry. 

Henri  Rajenbach  (Thomson-CSF,  Boulogne-Bil- 
lancourt,  France)  presented  preliminary  results  of 
,  hole  fixing  in  cooled  BSO  (B^SiOjo),  which  resulted 
in  the  ability  to  continuously  read  out  images  for 
several  hours  without  erasure.  This  is  possible 
because  a  photogenerated  hologram  written  by  the 
}  .electrons  is  compensated  by  the  holes,  «md  at  low 
temperatures  the  hole  mobility  is  much  lower  than 
the  electron  mobility,  so  that  a  hologram  written  as 
spatial  modulations  of  the  hole  density  remains 
frozen  into  the  crystal. 

,  Fai  Mok  (Northiup  Corp.  Research  and  Technol¬ 
ogy  Center,  Palos  Verdes,  CA)  presented  results  of 
multiple-image  storage  in  LiNb03:Fe,  which  used 
thermal  fixing  and  exposure  scheduling  to  compen¬ 
sate  for  erasure  during  writing  and  also  eliminated 
erasure  during  readout.  He  has  successfully  stored 
and  retrieved  more  than  500  images  of  more  than 
■  60,000  pixels  each,  with  greater  than  0.01%  diffrac¬ 
tion  efficiency. 

Dana  Anderson  (OCS  Center,  CU-Boulder) 
showed  winner-take-all  behavior  in  sequential  reccdl 
d)mamics  in  multicrystal  photorefractive  resona¬ 
tors.  The  computation  dynamics  inherent  to  the 
photorefractive  circuits  can  be  applied  as  a  very 
general  technique  for  obtaining  almost  any  dynam¬ 
ics,  provided  one  has  a  sufficient  number  of  modes 
and  sufficient  control  over  the  mode  interactions. 
Furthermore,  this  class  of  optical  system  is  one  of 
the  few  kinds  of  physical  systems  that  can  embed 
continuous  distributions  of  neural  activity  process¬ 
ing  in  continuous  time. 

Art  Gmitro  (University  of  Arizona,  Tucson,  AZ) 
and  H.  J.  White  (British  Aerospace,  Bristol,  UK) 
presented  analysis  of  the  limitations  of  space- 
variant-weighted  interconnections  using  e-beam 
written  computer-generated  holograms  for  inter¬ 
connecting  neuron  arrays  on  liquid-crystal  light 
valves.  They  both  concluded  that  up  to  about  64  x 
64'neuron  arrays  could  be  globally  interconnected  to 
implement  a  norudaptive  optical  neural  network 
using  this  technique.  However,  larger  networks 
would  be  beyond  the  technological  capabilities  of 
this  technology.  This  is  because  the  size  and  resolu¬ 


tion  requirements  of  a  space-variant  interconnec¬ 
tion  hologram  for  an  N  x  N-neuron  array  grow  as 
KN^  X  K?^,  where  K  is  an  oversampling  factor, 
typically  greater  than  10.  In  addition,  the  accuracy  of 
these  weighted  interconnects  might  be  too  low  for 
some  applications  (such  as  neural  optimizers),  but  it 
might  be  sufficient  for  optical  associative  memory. 

Alan  Yamamura  (CalTech)  presented  new  results 
of  writing  computer-generated  holograms  on  Sony's 
sampled-format  optical  disks.  Holographically  re¬ 
constructed  weight  matrices  were  written  onto  opti¬ 
cally  programmed  VLSI  neural  chips  to  control  the 
electronic  interconnect  topology  from  the  optical 
domain.  This  allows  rapid  reprogrammabilify  of  the 
interconnections  to  implement  multilayer  networks. 

Optical  neuron  technology 


Optical  neurons  need  to  sum  a  huge  number  of 
weighted  inputs  and  produce  a  threshold  output, 
The  goals,  stated  in  an  introduction  by  Demetri 
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FIGURE  3.  Polarization-based  optical  connectionist  machine 
has  learned  from  sunspot  pattern. 

Psaltis,  are  to  produce  devices  with  more  than  10,000 
neurons  with  a  response  time  in  the  microsecond 
range,  gain  greater  than  10,  and  power  dissipation 
below  a  milliwatt.  Although  such  a  device  capabili¬ 
ty  is  not  yet  available,  several  technologies  appear  to 
be  approaching  this  goal. 

Ferroelectric  liquid  crystals  are  promising  candi¬ 
dates  for  implementing  large  arrays  of  low-power 
neurons.  Garret  Moddel  (OCS  Center,  CU-Boulder) 
illustrated  the  capabilities  of  an  amorphous-silicon 
ferroelectric-liquid-crystal  (FLC)  optically  addressed 
SLM  (OASLM)  using  smectic  C*  and  smectic  A* 
materials.  The  smectic  A*  device  achieves  as  low  as 
a  4-n.s  response  time  for  a  1000  x  1000  array 
requiring  0.1-pJ/pixel  switching  energy.  Tliese  capa¬ 
bilities  make  this  an  almost-ideal  device  for  imple¬ 
menting  optical  neurons.  Further  improvements 
can  be  made  by  including  a  third  terminal,  which  his 
group  recently  demonstrated,  thereby  implement¬ 
ing  a  variable  threshold  device. 
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Tim  Drabik  (Georgia  Imtitute  of  Technology, 
Atlanta,  GA)  presented  recent  results  on  a  silicon 
VLSI  chip  coated  with  an  FLC,  which  incorporated 
photodetectors  and  acted  as  an  OASLM.  This  device 
nuiy  have  applications  as  an  optical  neuron  array  or 
as  an  early  vision  optical  preprocessor. 

Another  promising  approa^  is  to  use  optoelec¬ 
tronic  GaAs  modulators  and  sources  such  as  LEDs 
and  diode  lasers  driven  bj'  electronic  photodetec¬ 
tors.  Several  variants  were  proposed  for  optical 
neurons,  and  their  properties  were  compared  and 
contrasted.  Tony  Lentine  (AT&T  Bell  Labs,  Naper¬ 
ville,  IL)  showed  results  of  a  1.5-pJ  switching- 
energy  symmetric  self-electro-optic-effect  device.  Ar¬ 
rays  of  those  devices  can  implement  loser-take-all 
dynamics  by  electrically  intercormecting  them  in 
series.  Although  this  is  the  opposite  of  the  winner- 
take-all  dynamics  required  in  many  neural  models,  it 
can  be  modified  by  following  with  a  separate  optical 
inversion  stage. 

Demetii  Ps^tis  showed  results  of  a  10  x  10  array 
of  phototiansistor  Dariington-amplifier  LED  neurons 
fabricated  in  GaAs.  LEDs  were  chosen  instead  of 
more-efficient  diode  lasers  because  they  have  no 
current  threshold,  and  the  power  dissipation  of  a 
very  large  array  of  optical  neurons  can  be  minimized. 

Implications  of  neural-network  theory 
Neural-network  theory  has  several  implications  for 
the  requirements  and  limitations  of  optical  neural 
networks.  These  issues  were  discussed  at  length 


during  the  workshop.  Two  specific  consequences  are 
mentioned  here. 

Santosh  Venkatesh  (University  of  Pennsylvania) 
presented  results  on  the  required  dynamic  range  of 
synapses  and  showed  that,  with  binary  syitapses, 
storage  capacity  is  only  reduced  by  a  factor  of  two 
from  real  synapses.  He  presented  a  simple  directed- 
drift  leartung  ^gorithm  for  these  binary  synapses 
and  concluded  that  low-dynamic-range  optical  syn¬ 
apses  nuy  be  adequate  for  many  applications.  This, 
however,  did  not  alleviate  the  dynajiuc-range  re¬ 
quirements  of  the  neurons,  nor  did  it  address  the 
issue  of  contrast  ratio  or  noise. 

Eric  Baum  (NEC  Research  Institute)  discussed  the 
limitations  mherent  to  learning  from  examples  and 
concluded  that  the  huge  number  of  interconnects 
available  in  optical  systems  may  require  a  huge 
number  of  training  exemplars  to  achieve  valid 
generalization.  One  suggestion  to  alleviate  the  unfa¬ 
vorable  scaling  properties  of  large  networks  was  to 
expand  or  prune  the  networks  to  fit  the  task.  He 
suggested  ^at  the  superabimdance  of  resources  in 
optics  may  allow  fast  learning,  followed  by  pruning, 
to  achieve  a  miivimal  optical  network  quickly  and 
obtain  good  generalizahon. 

Kelvin  Wagner 
University  of  Colorado  at  Boulder 
Lee  Giles 
NEC  Resmrch  Institute 
bemetri  Psaltis 
California  Institute  of  Technology 
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